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I.  INTRODUCTION 


A.  PURPOSE 

The  purpose  of  this  thesis  is  to  examine  the  current  state  of  Internet  usage  by 
Information  Technology  Management  (ITM)  students  at  the  Naval  Postgraduate  School 
(NPS)  in  Monterey,  California.  The  current  state  of  usage,  for  the  purposes  of  this  thesis, 
consists  of  the  student’s  ability  to  use  the  Internet  along  with  the  current  instructional 
sources  which  teach  the  students  this  ability. 

ITM  students  will  become  DoD’s  information  technology  managers  of  the  future. 
It  is  assumed  that  there  is  some  advantage  to  having  DoD  IT  managers  who  are  well- 
versed  in  using  the  Internet.  This  thesis  will  examine  these  theoretical  advantages  of 
Internet  usage,  and  help  ensure  that  ITM  students  are  being  prepared  to  use  the  Internet 
in  the  future. 

This  thesis  will  not  solely  benefit  ITM  students,  however.  All  graduates  from  NPS 
should  be  literate  in  the  use  of  the  Internet.  The  results  of  this  study  will  be  applicable 
to  all  graduates  of  NPS  as  well  as  other  Internet  users  through  out  DoD. 

B.  RESEARCH  QUESTIONS 

There  are  two  primary  research  questions  in  this  thesis.  One  looks  at  the  ability 
of  ITM  students,  while  the  other  looks  at  the  current  source  of  Internet  instruction. 

Is  there  a  relationship  between  experience  and  the  ability  to  use  the  Internet 
effectively?  It  would  seem  obvious  that  the  ability  of  a  student  to  perform  a  given  task 
would  be  proportionate  to  the  experience  or  knowledge  level  of  that  student.  This 
research  question  will  confirm  this  fact.  It  will  also,  however,  provide  a  measure  of 
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ability  at  various  points  on  an  experience  timeline.  This  should  be  useful  in  determining 
methods  to  improve  the  learning  process,  if  necessary. 

What  instructional  source  has  taught  ITM  students  to  use  the  Internet?  In  other 
words,  where  are  the  students  learning  how  to  use  the  Internet?  The  students  may  be 
learning  from  three  different  categories  of  sources.  They  may  be  learning  from  informal 
sources  such  as  their  fellow  students  or  by  exploring  on  their  own.  They  could  be 
learning  from  formal  instruction  such  as  class  assignments,  or  they  could  be  learning  from 
semi-formal  sources  such  as  computer  center  classes  or  personnel.  This  is  of  interest, 
especially  if  the  prior  research  question  shows  that  changes  are  necessary. 

C.  SCOPE 


This  thesis  will  consist  of  an  observation  of  a  sample  set  from  the  ITM  student 
population  to  answer  the  primary  research  questions.  Five  students  from  each  of  the  four 
present  ITM  sections,  PM33,  PM41,  PM43,  and  PM51,  will  be  used  as  the  sample  set. 
Students  will  be  observed  performing  a  series  of  tasks  using  their  Unix  accounts  in  one 
of  the  NPS  computer  center  labs.  Their  ability  to  perform  the  task  will  be  noted.  The 
time  required  will  also  be  recorded.  The  methodology  of  this  observation  process  is 
discussed  in  much  greater  detail  in  Chapter  II. 

The  information  gathered  in  these  observations  will  then  provide  the  basis  for  the 
conclusions  and  recommendations  made  in  this  thesis. 

D.  LITERATURE  REVIEW 

1.  Learning  Theory 

A  short  overview  of  learning  theory  and  its  application  to  computer  based  tasks 
is  important  to  the  overall  understanding  of  why  the  students’  performance  did  or  did  not 
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meet  the  expectations  of  the  researcher.  It  is  also  important  to  the  development  of  ways 
to  improve  upon  current  instructional  conditions. 

There  are  two  parts  to  describing  the  learning  process.  First,  there  is  the  type  of 
knowledge  with  which  we  are  dealing,  and  second,  there  is  the  process  of  knowledge 
instruction 

2.  Knowledge  Types 

There  are  three  basic  categories  of  knowledge:  declarative,  procedural,  and  self- 
knowledge.  (Dillon  &  Sternberg,  1986) 

Declarative  knowledge  is  the  raw  facts  associated  with  a  subject.  It  is  usually  a 
hierarchial  elaboration  on  a  given  subject,  with  the  most  detailed  information  contained 
at  the  very  bottom  of  the  structure.  There  is  no  need  to  use  the  knowledge,  only  to  be 
able  to  present  it.  An  example  of  declarative  knowledge  would  be  the  rote  recital  of  dates 
on  a  history  examination. 

Procedural  knowledge  is  knowledge  which  gives  the  ability  to  perform  some 
function.  It  is  the  natural  follow-on  to  declarative  knowledge.  Procedural  knowledge 
requires  more  sophisticated  reasoning  on  the  part  of  the  user,  as  the  user  must  develop 
different  sets  of  logic  for  different  tasks. 

Self-knowledge  is  an  awareness  of  one’s  reasoning  process.  It  involves  accessing 
the  logic  used  in  procedural  tasks  and  examining  it  for  strengths  and  weaknesses.  This 
is  the  most  advanced  form  of  knowledge. 

Procedural  knowledge  was  most  relevant  for  this  study.  It  allowed  the  students 
to  perform  the  given  tasks  on  the  Internet. 

3.  Knowledge  Instruction 

There  are  four  phases  to  knowledge  instruction  (Alessi  &  Trollop,  1991): 
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1.  Presenting  information. 

2.  Initial  acquisition. 

3.  Fluency  and  retention. 

4.  Assessment. 

Presenting  information  and  initial  acquisition  normally  take  place  at  the  same  time. 
In  a  classroom,  for  example,  an  instructor  presents  the  information  at  the  same  time  as 
a  student  begins  to  retain  it.  This  can  be  accomplished  either  by  classroom  instruction 
or  by  tutorials.  This  corresponds  to  the  learning  of  declarative  knowledge. 

Fluency  and  retention  require  the  use  of  drills  which  cause  a  student  to  work 
through  a  problem  on  his  or  her  own.  Success  provides  the  procedural  knowledge  that 
students  must  use  to  complete  the  tasks  in  this  thesis. 

The  last  area  of  knowledge  instruction  is  the  assessment  or  testing  of  ability.  This 
provides  validation  to  both  the  instruction  method  and  the  student’s  ability. 

In  general,  computer  based  learning  is  an  automated  application  of  the  techniques 
already  in  use  manually,  especially  tutorials.  The  current  advantage  of  computer  usage 
is  increased  speed  and  attention-holding  capability.  (Ngwenyama,  1993) 

The  phases  of  instruction  of  most  interest  to  this  study  are  fluency  and  retention, 
and  assessment.  Assessment  will  help  determine  the  current  state  of  the  ITM  student 
population  and  fluency  and  retention  will  help  develop  ways  to  improve  it. 

4.  Current  Benefits  of  Internet  Usage 

The  benefits  of  Internet  usage  are  central  to  why  this  study  was  conducted. 
Therefore,  we  must  look  at  what  the  Internet  is  and  why  its  use  by  ITM  students  is 
important. 

The  Internet  is  the  largest  computer  network  in  the  world,  primarily  supported  by 
government  grants.  The  origin  of  the  Internet  was  the  DoD’s  Advanced  Research  Projects 
Agency  Network  (Arpanet)  begun  in  the  late  1960’s.  In  1983,  the  Transmission  Control 
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Protocol/Intemet  Protocol  (TCP/IP)  was  made  mandatory  by  the  Defense  Communications 
Agency  for  both  the  Arpanet  and  the  Milnet,  a  DoD  network.  After  this  point, 
commercial  enterprises  began  to  produce  products  to  this  standard.  The  collection  of  over 
17,000  networks  that  now  share  this  protocol  is  known  as  the  Internet.  The  largest 
backbone  of  the  Internet  is  now  the  National  Science  Foundation’s  NSFnet  with  over 
4,000  networks  linked  to  it.  In  1992,  the  Internet  had  over  1  million  host  computers  with 
many  millions  of  users. (Perry,  1992) 

The  benefit  of  using  the  Internet  to  find  and  retrieve  information  is  a  more 
productive  group  of  students.  Students  with  an  ability  to  access  information  on  the 
Internet  are  not  tied  to  geographic  locations,  such  as  an  institution  library.  Nor  are  they 
excluded  from  access  to  information  due  to  academic  position  or  standing.  (Watson,  1994) 
However,  using  the  Internet  is  not  intuitive.  Since  the  Internet  is  decentralized, 
there  are  no  common  standards  and  no  formal  advertisement  of  information  locations. 
The  Internet  is  far  too  large  for  the  imposition  of  a  hierarchical  structure.  (Schwartz, 
1993) 

As  an  example  to  illustrate  the  need  to  provide  efficient  locating  capabilities  for 
Internet  information,  in  December  1991,  the  University  of  Colorado  conducted  a  study 
involving  the  monitoring  of  File  Transfer  Protocol  (FTP)  traffic.  During  the  course  of  the 
study  (13  days),  39,324  files  were  transferred.  However,  this  was  only  44  percent  of  the 
actual  FTP  traffic.  The  remaining  traffic  was  directory  requests,  or  in  other  words, 
individuals  looking  around  for  information.  (Schwartz,  1993) 

What  is  needed  is  a  set  of  services  which  allow  the  user  to  search  and  retrieve  the 
various  information  available  on  the  Internet.  This  will  allow  students  to  achieve  the  full 
potential  of  Internet  usage.  (Obraczka  et  al,  1993) 


5 


E.  BASIC  INTERNET  TOOLS 


As  previously  discussed,  the  decentralized  nature  of  the  Internet  means  that 
information  may  be  found  at  many  different  sites,  and,  possibly,  in  several  different 
formats.  The  location  of  information  may  or  may  not  be  published,  or  known  to  the  user. 
Users  are  often  required  to  use  a  variety  of  tools  to  search  for  and  acquire  desired 
information,  such  as  text  files,  current  research  material,  and  colleagues  e-mail  addresses. 
An  understanding  of  theses  tools  is  needed  prior  to  conducting  the  observation  of  ITM 
students  attempting  to  perform  various  Internet  tasks.  (Schwartz,  1993) 

Table  1  shows  a  basic  set  of  tools  that  may  be  used  in  the  Internet.  It  is  not 
intended  to  be  an  exhaustive  list,  but  instead  as  an  overview  of  current  capabilities. 

Electronic  Mail.  Simple  Mail  Transfer  Protocol  (SMTP)  allows  the  user  to  send 
mail  across  the  Internet  regardless  of  the  user’s  own  particular  file  system.  The  Internet 
provides  a  series  of  relay  mail  servers  which  use  the  mail’s  Internet  Protocol(IP)  address 
as  a  means  to  determining  the  mail’s  ultimate  destination.  Mail  is  normally  short 
messages  in  ASCII  text.  ASCII  text  is  generic  text  that  has  been  stripped  of  program 
specific  control  characters.  (Postel,  1982) 

Netfind.  The  use  of  electronic  mail  relies  on  knowledge  of  a  recipient’s  e-mail 
address.  Netfind  locates  unknown  e-mail  addresses.  It  uses  the  search  mechanisms 
available  within  each  individual  domain’s  network  mail  server  rather  than  a  common 
format  that  would  be  needed  for  a  central  database. 

To  begin  a  Netfind  search,  a  user  provides  a  desired  person’s  name  and  a  short 
group  of  keywords  describing  the  possible  location  of  the  desired  person.  An  example 
of  such  a  string  could  be  "smith  navy  mil."  Netfind  then  uses  this  information  in  a  seed 
database  which  contains  a  listing  of  possible  domain  matches.  It  then  asks  a  user  to  pick 
the  domains  he  or  she  desires  to  search.  In  the  example,  Netfind  may  return 
"nps.navy.mil",  "usna.navy.mil",  and  nosc.navy.mil."  Once  the  user  chooses  a  desired 
domain,  Netfind  queries  the  SMTP  server  as  to  all  possible  mail  forwarding  information 
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Internet  Tool 

Electronic  Mail  (E-Mail) 
Netfind 

FTP  (File  Transfer  Protocol) 
Gopher 

Veronica 

WAIS  (Wide  Area  Information 
Servers) 

Telnet 

WWW  (World  Wide  Web) 

Web  Robots 


Table  1.  Basic  Int 
After 


Use 

Electronic  communication 
Locate  E-Mail  addresses 
Transfer  of  files 

A  public  access  distributed  document 
retrieval  system 

Keyword  searching  of  document  titles 
at  Gopher  sites 

Full-text  searching  of  distributed 
documents 

Access  to  remote  machines 

A  public  access  distributed  hypertext 
and  multimedia  document  retrieval 
system 

Keyword  searching  of  pages  at  WWW 
sites 


iet  Tools  and  their  Uses 
atson,  1994 
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on  each  machine  at  that  domain.  Netfind  uses  the  "finger"  command  to  return  all  possible 
e-mail  addresses  for  the  desired  individual  on  each  machine. 

The  seed  database  constantly  updates  itself  from  a  variety  of  e-mail  monitoring 
and  reporting  services.  Since  1993,  Netfind  could  search  over  9,000  sub-domains, 
containing  the  e-mail  addresses  of  approximately  5.5  million  users.  This  includes  191 
sub-domains  in  the  ".mil"  domain.  (Schwartz,  1993) 

FTP.  File  Transfer  Protocol  allows  a  user  to  transfer  files  between  two  computers. 
The  most  common  use  on  the  Internet  is  anonymous  FTP.  In  anonymous  FTP,  the  user 
has  access  to  a  computer  system  without  having  an  account.  This  allows  any  user  to 
obtain  a  large  amount  of  information,  without  the  usual  account  overhead.  (Rovers,  1994) 
Gopher.  Gopher  is  a  hierarchical  structure  that  allows  users  to  retrieve  distributed 
information  on  the  Internet.  Gopher  organizes  information  into  a  series  of  directories  and 
sub-directories,  with  the  lowest  level  being  the  actual  files.  Users  navigate  through 
gophers  using  a  hierarchical  tree  structure.  (Obraczka  et  al,  1993) 

-Veronica.  Veronica  does  keyword  searches  on  a  database  of  titles  of  gopher  items, 
both  files  and  directories.  The  Veronica  service  performs  its  function  in  two  parts.  First, 
menu  data  for  constructing  the  database  is  gathered  from  registered  gopher  servers.  The 
listing  of  registered  gopher  servers  is  maintained  at  the  University  of  Minnesota.  Second, 
Veronica  allows  users  to  conduct  keyword  searches  on  this  database.  These  two  functions 
are  not  necessarily  performed  by  the  same  machine;  the  database  may  be  obtained  from 
a  common  repository.  (Foster  &  Barrie,  1993) 

WAIS.  Wide  Area  Information  Servers  is  a  full-text  information  retrieval 
architecture.  It  works  by  having  a  central  "Directory  of  Services"  which  points  a  user  to 
the  database  that  contains  the  information  they  desire. 

To  conduct  a  WAIS  search,  a  user  prepares  a  query  which  is  transmitted  in  the 
WAIS  protocol  to  the  "Directory  of  Services,"  or  yellow  pages,  which  is  installed  on  a 
known  WAIS  server.  The  "Directory  of  Services"  forwards  the  query  to  all  relevant 
databases,  based  on  its  knowledge  of  what  these  databases  contain.  A  specific  database 
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then  returns  the  user  a  ranked  listing  of  all  objects  which  contain  the  desired  text  string. 

Each  individual  database  is  responsible  for  providing  updated  content  information 
to  the  central  yellow  pages.  Currently  there  are  over  300  databases  referenced  by  the 
"Directory  of  Services."  (Obraczka  et  al,  1993) 

World  Wide  Web.  The  World  Wide  Web  was  developed  at  CERN  (European 
Laboratory  for  Particle  Physics)  in  Switzerland.  It  uses  a  distributed  structure  supported 
by  a  graphical  user  interface(GUI)  using  hypertext  techniques.  Data  is  organized  into  text 
objects  which  are  linked  to  a  series  of  hypertext  cover  sheets  or  index  objects  known  as 
Web-pages.  (Obraczka  et  al,  1993) 

Web  Robots.  A  Web  Robot  explores  the  Internet  autonomously  to  find  new 
information.  It  does  this  by  starting  with  a  known  WWW  document  and  identifying  new 
places  to  explore  by  looking  at  the  links  from  that  document  to  other  documents.  It  then 
repeats  the  process  indefinitely,  while  compiling  indices  of  the  information  it 
encounterers.  When  a  Web  Robot  is  sent  a  query  by  a  user,  it  uses  these  indices  to 
provide  a  listing  of  relevant  WWW  sites.  (Pinkerton,  1995) 

For  convenience,  these  tools  can  be  divided  into  three  categories: 

1 .  Those  which  allow  the  Internet  user  to  communicate,  such  as  e-mail. 

2.  Those  that  allow  the  user  to  transfer  or  view  information,  such  as  FTP,  Gopher 

or  WWW. 

3.  Those  that  allow  the  user  to  find  information,  such  as  Netfind,  Veronica,  or 

WebCrawlers. 

The  most  complex  tools  are  the  ones  associated  with  the  finding  of  information. 

With  a  working  knowledge  of  this  basic  set  of  tools,  the  average  Internet  user 
should  be  able  to  find  and  access  a  reasonable  portion  of  the  available  information  on  a 
given  subject.  However,  no  published  articles  or  studies  were  found  as  to  what  portion 
of  the  total  available  information  any  one  tool  was  actually  able  to  locate. 
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H.  METHODOLOGY 


The  methodology  of  this  thesis  is  an  observation  of  students  performing  a  set  of 
tasks.  The  organization  of  this  methodology  is  derived  from  Emory  &  Cooper,  1991. 

A.  DESIGN  STRATEGY 

This  thesis  portrays  the  ability  of  Information  Technology  Management  (ITM) 
students  to  use  the  Internet  effectively.  The  desired  data  should  provide  both  a  concrete 
measure  of  each  section’s  ability  and  the  means  for  comparison  between  the  sections.  A 
section  is  an  administrative  designation  used  at  NPS  to  ,  denote  a  group  of  students 
curriculum  and  arrival  date.  For  example,  PM33  is  the  ITM  section  which  arrived  in  the 
third(3)  quarter  of  fiscal  year  1993.  The  following  set  of  hypotheses  will  be  used  as  a 
basis  for  the  observations  conducted: 

Hypothesis  1 .  The  ability  to  perform  various  tasks  using  the  Internet  and  student 
Unix  accounts  improves  with  experience,  experience  being  defined  as  time  at  the  Naval 
Postgraduate  School. 

Hypothesis  2.  The  prime  source  of  the  students’  ability  to  use  the  Internet  and 
their  Unix  accounts  is  not  through  a  formal  source,  such  as  class  assignments,  computer 
center  lectures,  or  computer  center  personnel,  but  rather  through  an  informal  method  of 
sharing  knowledge  among  themselves. 

A  set  of  direct  behavioral  observations  was  chosen  to  test  hypothesis  1.  Direct 
observation  was  chosen  in  view  of  the  observer’s  limits  in  time,  money  and  hardware. 
An  unobtrusive  method  such  as  automatic  monitoring  of  activities  within  a  student’s  Unix 
account  would  have  provided  better  data,  but  this  method  proved  physically  unpracticable 
within  the  scope  of  this  study. 

Three  categories  of  tasks  were  used  as  a  representative  measure  of  ability  as 
contained  in  hypothesis  1.  These  three  categories  were  communications,  data  transfer  and 
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knowledge  acquisition.  Time  required  to  perform  the  task  was  used  as  a  measure  of 
effectiveness  in  each  case.  Although  the  quickest  answer  does  not  necessarily  provide  the 
best  answer,  it  was  the  most  suitable  measure  in  this  case.  This  choice  of  measurement 
is  discussed  further  below. 

a.  Communications:  Was  the  subject  able  to  communicate  effectively  using  e- 
mail,  including  the  acquiring  of  various  unknown  e-mail  addresses? 

b.  Data  Transfer:  Given  the  location  of  specific  pieces  of  data,  could  the  subject 
retrieve  them  using  a  variety  of  tools,  such  as  file  transfer  protocol  (FTP),  gopher  and  the 
world  wide  web  (WWW)? 

c.  Knowledge  Location:  Given  a  specific  topic  or  site,  could  the  subject  locate 
the  requested  information? 

Hypothesis  2  would  be  tested  in  the  second  portion  of  the  observation,  using  a 
short  series  of  demographic  questions. 

B.  MEASUREMENT 

Along  with  discussing  the  reasons  for  the  various  scales  used  within  the 
observation,  this  section  looks  at  the  validity,  reliability,  practicality,  and  error  sources 
associated  with  the  measurements  used. 

1.  Scale 

A  ratio  scale  was  used  for  the  observation  of  the  tasks  associated  with  hypothesis 
1 .  As  discussed  in  the  design  strategy,  the  amount  of  time  required  to  perform  a  given 
task  was  recorded.  Any  task  requiring  more  than  five  minutes  was  scored  as  "unable  to 
perform  the  task."  The  time  taken  was  recorded  by  the  observer. 

An  ordinal  scale  was  used  for  hypothesis  2.  Students  were  asked  to  rank  the  top 
three  sources  from  which  they  had  gained  the  knowledge  they  had  used  in  the  task-based 
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portion  of  the  observation.  Although  this  ordinal  scale  will  not  allow  parametric  testing 
of  this  data,  it  is  the  most  appropriate  form  in  which  to  measure  this  data.  Independent 
tracking  of  a  source  of  knowledge  was  not  possible. 

2.  Validity 

Validity  is  the  ability  of  the  observation  to  measure  what  it  is  designed  to  measure. 

Content,  criterion,  and  construct  validity  are  the  primary  concerns. 

a.  Content  validity  is  the  degree  to  which  the  observation  covers  all  applicable 
areas  of  the  material  being  studied,  in  this  case  Internet  ability.  Based  on  the  literature 
review  and  the  pilot  test,  there  seems  to  be  good  content  validity.  The  three  categories 
chosen  to  test  hypothesis  1,  communications,  data  transfer,  and  knowledge  acquisition, 
appear  to  provide  adequate  coverage  of  the  tasks  normally  required  of  a  student  Internet 
user.  Hypothesis  #2  has  a  very  narrow  set  of  possible  sources  as  to  where  the  student 
acquired  his  or  her  skill  level.  The  ordinal  questions  chosen  should  provide  a  complete 
set  of  possible  answers. 

b.  Criterion  validity  is  how  well  the  chosen  measure,  in  this  case  time,  represents 
the  actually  state  of  what  is  being  measured.  Using  time  as  a  measure  for  hypothesis  #1 
is  not  the  only  possible  choice.  A  measure  of  the  quality  of  the  data  delivered  per  task 
may  have  been  viable,  or  an  analysis  of  the  technique  and  thought  processes  used  may 
also  have  been  of  value.  However,  time  is  the  most  useful  measure  available  because  it 
provides  the  ability  to  conduct  statistical  analysis  plus  it  eliminates  problems  with 
observer  interpretation. 

c.  Construct  validity  is  the  degree  to  which  a  measure  conforms  to  other 
theoretical  work.  In  this  case,  there  were  no  other  studies  with  which  to  compare.  The 
tasks  the  subjects  are  asked  to  perform  are  considered  basic  and  fundamental  to  using  the 
Internet,  therefore,  the  subjects  ability  to  perform  these  tasks  is  a  valid  measure  of  actual 
ability  and  should  have  good  construct  validity. 
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3.  Reliability 


Reliability  is  the  degree  to  which  the  observation  provides  consistent  results. 
There  are  three  types  of  reliability  which  are  of  concern:  stability,  equivalence,  and 
internal  consistency. 

a.  If  a  subject  scores  the  same  on  repeated  observations  then  there  is  stability  in 
the  measurement.  In  this  thesis,  there  were  no  retests  of  subjects.  Therefore  there  is  no 
actual  means  to  test  the  stability  of  the  measured  results.  However,  there  appears  to  be 
no  reason  that  the  measure  is  not  stable.  Situational  errors  may  have  occurred,  as 
discussed  below,  but  they  are  a  function  of  observation  environment  rather  than  the  tool 
itself. 

b.  Equivalence  considers  whether  different  observers  will  achieve  different  results. 
Again,  there  is  no  means  to  actually  confirm  equivalence  in  the  observation  since  there 
was  only  one  observer.  The  observer  kept  interaction  with  the  subjects  to  a  minimum  and 
only  recorded  objective  quantities  such  as  time.  A  second  observer  should  be  able  to 
reproduce  similar  results. 

c.  Internal  Consistency  ensures  that  similar  tasks  with  one  subject  will  provide 
similar  results.  There  was  no  attempt  to  test  internal  consistency  and  there  is  no  way  to 
verify  that  it  exists  other  than  to  note  whether  subjects  who  performed  poorly  in  certain 
portions  of  the  observation  continued  to  perform  poorly  and  those  who  performed  well 
continued  to  perform  well. 

4.  Practicality 

Practicality  was  a  major  design  consideration  due  to  the  previously  discussed 
shortcomings  in  resources,  mainly  time.  The  observation  was  easily  carried  out  by  a 
single  observer  using  a  public  lab  in  the  course  of  one  work  week.  All  tasks  are  straight 
forward  with  a  single  possible  correct  answer,  while  allowing  individual  subjects  to  use 
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a  variety  of  techniques  to  reach  the  answer.  The  scales  used  provide  a  simple,  useful  data 
set. 


5.  Error  Sources 

There  were  several  opportunities  for  errors  to  have  been  introduced  into  the 
measurement  process.  However,  these  errors  are  consistent  between  sections  of  the 
sample,  and  although  they  may  flaw  the  empirical  data,  the  relative  measures  of  the  data 
remain  valid. 

a.  Respondent  error  seems  to  be  minimal.  All  subjects  appeared  to  the  observer 
to  honestly  attempt  to  perform  all  tasks.  All  have  similar  backgrounds  with  regard  to 
career  and  social  class.  There  may  have  been  small  amounts  of  error  introduced  by 
student  anxiety  over  other  classes,  et  cetera,  but  this  did  not  appear  to  be  significant. 

b.  Situational  error  is  the  most  significant  source  of  error.  It  was  impossible  to 
tailor  the  lab  situation  to  each  subject,  therefore  there  were  multiple  sources  for  situational 
error.  Many  of  the  subjects  were  used  to  working  on  home  PC-based  interface  as 
opposed  to  a  Unix-based  workstation.  The  presence  of  other  students  in  the  lab  may  have 
affected  the  subjects  ability  to  concentrate  on  the  given  tasks.  The  presence  of  an 
observer  may  have  also  contributed  to  an  uncomfortable  atmosphere  in  which  to  conduct 
the  experiment.  All  these  factors  combined  to  give  an  advantage  to  subjects  who  were 
used  to  working  on  a  Unix  workstation  in  a  crowded  lab  environment. 

c.  The  measurer  was  a  minimal  error  source.  The  same  observer  performed  all 
observations.  The  personal  acquaintance  of  the  observer  to  any  of  the  subjects  did  not 
flaw  the  observation,  since  all  tasks  were  of  a  very  objective  nature. 

d.  Instrument  was  also  a  minimal  source  of  error.  The  instrument  is  simple  and 
has  been  shown  to  be  valid.  Both  the  pilot  test  and  the  actual  testing  showed  the  tasks 
to  be  clear  and  complete. 
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C.  SAMPLING  DESIGN 


Since  Hypothesis  1  required  various  experience  levels,  a  stratified  sample  of  the 
ITM  student  population  was  needed.  A  sample  of  five  students  from  each  of  the  four 
current  ITM  sections  was  chosen  as  the  sample  that  would  provide  the  needed  data.  This 
sample  size  was  chosen  based  on  the  small  total  population  size.  This  choice  is  addressed 
in  detail  in  the  discussion  of  sample  size. 

1.  Sample  Type 

The  sample  was  drawn  from  the  ITM  student  population  at  large.  It  was  a 
convenience  sample  in  that  the  observer  had  no  means  of  drawing  a  random  sample  from 
the  population,  mainly  because  there  was  no  way  the  observer  could  compel  participation 
from  the  randomly  selected  subjects.  Therefore,  the  observer  relied  on  volunteers  to 
provide  the  sample  set. 

2.  Sample  Size 

As  previously  discussed,  five  subjects  were  used  from  each  of  the  four  current 
ITM  sections.  This  was  necessitated  by  the  small  size  of  the  total  population.  The  total 
ITM  population  by  section  is  as  follows: 
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PM33 

27 

PM41 

53 

PM43 

28 

PM51 

57 

Table  2.  ITM  Population  by  Section 

Since  there  was  no  basis  for  an  interval  estimate,  the  effects  of  such  a  small 
sample  size  on  the  observation  could  not  be  determined  until  the  results  were  examined. 

Rather  than  begin  with  an  interval  estimate  to  determine  the  desired  sample  size, 
the  standard  deviation  of  the  sample  was  used  to  determine  the  interval  estimate  with  a 
0.95  confidence  level.  Using  the  standard  deviation  from  each  section’s  sample,  the 
interval  estimate  was  determined  as  follows: 


i= (1.96) — — 
sfn2 1 


where 

i  =  Interval  estimate  in  minutes, 
s  =  Standard  deviation  of  the  section  sample, 
n  =  Sample  size,  in  all  sections  this  number  was  5. 
1.96  =  Standard  errors  for  a  0.95  confidence  level. 
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This  equation  gives  interval  estimate  (i)  by  section  with  0.95  confidence  level  as  follows: 


s 

i 

PM33 

51  secs 

PM41 

0.909 

!  PM43 

50  secs 

PM51 

46  secs 

Table  3.  Interval  Estimate  for  Task  Observations 

Therefore,  with  a  0.95  confidence  level,  the  results  of  the  observation  of  the 
sample  set  are  within  53  seconds  of  the  population  as  a  whole  based  on  the  standard 
deviation  of  each  section’s  data  set  and  a  sample  size  of  five. 

By  comparison,  the  population  would  have  been  unable  to  support  the  observation 
had  interval  estimate  been  used  to  dictate  sample  size.  An  interval  estimate  of  30  seconds 
increases  the  required  sample  size  to  37  subjects  per  section.  Even  with  a  finite 
population  adjustment,  the  sample  size  remains  over  15  subjects  per  section.  Given  the 
population  size  this  was  highly  impracticable. 

Note:  Since  the  results  were  in  a  bipolar  distribution,  the  standard  deviation  used 
for  each  section  was  taken  only  from  the  positive  answers  and  not  the  "unable  to  perform 
task  responses.  That  is  the  observations  that  could  be  quantified  in  terms  of  time. 
Therefore,  the  sample  mean  used  to  calculate  the  standard  deviation  is  not  representative 
for  use  as  a  comparison  between  the  groups,  and  was  only  used  as  a  point  around  which 
to  measure  the  deviation  of  each  section. 
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3.  Sample  Validity 


Validity  is  also  important  in  the  sample  process.  In  this  case,  validity  is  how  well 
the  sample  set  represents  the  entire  population.  There  are  two  parts  to  sample  validity, 
accuracy  and  precision. 

a.  An  accurate  sample  will  have  no  systemic  variances  or  biases  associated  with 
it.  The  use  of  a  non-random  sample  could  be  a  source  of  systematic  variance.  However, 
in  this  case  the  standard  deviations  were  low  enough  to  make  this  unlikely. 

b.  Precision  is  a  reflection  of  how  well  the  effects  of  chance  have  been  removed 
from  the  sample  set.  This  usually  comes  into  play  when  all  other  systemic  variances  have 
been  accounted  for.  Again,  the  low  standard  deviation  of  the  sample  sets  shows  that  there 
is  good  precision. 

D.  THE  PILOT  TEST 

A  pilot  test  of  the  observation  tasks  was  conducted  prior  to  the  actual  observation 
of  sample  subjects.  The  pilot  test  was  intended  to  show  errors  in  both  the  observation 
tasks  and  the  observer’s  technique.  The  pilot  test  procedures  were  identical  to  the  actual 
observations  in  controllable  respects,  such  as  location,  tasks  and  equipment  used.  The 
personal  relationship  between  the  subjects  and  the  observer  provided  the  only  possible 
source  of  error  in  this  procedure. 

Three  volunteer  subjects  were  chosen  from  PM33,  the  senior  section.  The 
volunteers  were  chosen  based  on  the  observer’s  opinion  that  they  provided  a  valid  cross- 
section  of  the  PM33  population.  There  was  no  attempt  to  maintain  randomness  in  the 
selection  of  pilot  test  subjects. 

Although  the  pilot  test  subjects  showed  varied  levels  of  competence  on  some  tasks, 
their  results  within  each  category  were  very  similar.  This  indicated  that  the  initial  group 
of  tasks  should  be  sufficient  to  provide  a  representation  of  each  section  as  a  whole. 
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In  post-observation  discussion,  all  three  pilot  test  subjects  indicated  that  the  tasks, 
in  their  opinions,  provided  a  fair  test  of  their  abilities  to  perform  the  three  categories  of 
tasks.  They  also  anticipated  a  difference  in  results  between  sections  due  to  their  own 
experience  in  learning  to  perform  the  observed  tasks. 

E.  OBSERVATION 

The  observation  was  conducted  using  a  Sun  Sparc  10  workstation  in  the  public 
computer  lab  in  Root  Hall  at  the  Naval  Postgraduate  School,  Monterey,  California.  The 
subjects  used  their  own  accounts  and  were  allowed  to  use  any  techniques  or  tools  that 
were  available. 
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HI.  OBSERVATION  RESULTS 


This  chapter  is  a  summary  of  the  data  collected  during  the  task  observations. 
Although,  in  some  cases,  this  is  not  the  raw  data,  all  cumulative  data  presented  are  taken 
directly  from  the  raw  data.  In  all  tables,  represents  a  ’0’  for  ease  of  visual  inspection. 
The  remaining  raw  data  are  available  in  Appendix  B.  An  analysis  of  these  data  is 
provided  in  Chapter  IV. 

A.  DEMOGRAPHICS 

The  following  data  provide  the  demographic  information  that  was  gathered  about 
the  subjects  during  the  observation.  The  demographics  concern  solely  the  characteristics 
of  the  subjects’  Unix  account  usage.  No  personal  demographics,  such  as  age,  sex  or 
military  community,  were  recorded  during  the  observation. 

Table  4  is  the  response  to  a  direct  question  as  to  how  often  a  subject  logs  in  to 
his  or  her  Unix  account.  The  question  was  posed  by  the  observer  to  the  subjects.  It  is 
the  subject’s  own  answer,  without  independent  verification.  The  sample  for  each  section 
was  five  subjects. 


PM33 

PM41 

PM43 

PM51 

<  Once/Week 

- 

- 

- 

- 

Once/Week 

- 

- 

3 

- 

2-3/Week 

- 

- 

1 

- 

4-5/Week 

- 

- 

- 

- 

6-7/Week 

1 

- 

1 

- 

>  Once/Day 

4 

5 

- 

5 

Table  4.  Login  Frequency 
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The  data  for  Table  4  were  not  independently  gathered  or  confirmed.  There  is  no 
confidence  factor  or  interval  estimate  computed  for  this  data.  The  data,  however,  still 
provide  insights  into  the  usage  patterns  of  ITM  students.  The  interpretation  of  these  data 
will  be  discussed  in  more  detail  in  Chapter  IV. 

Table  5  shows  the  extent  to  which  a  subject  customized  his  or  her  computer  center 
account  workspace.  This  was  recorded  by  the  observer  without  input  from  a  subject.  In 
this  case,  customization  is  defined  as  any  modification  to  the  standard  Unix  X-windows 
workspace  that  allows  a  subject  to  function  more  effectively.  In  Table  5  "yes"  means  the 
account  was  customized,  and  "no"  means  the  account  was  not. 

A  hotlist,  as  used  in  Table  5,  is  a  personalized  list  of  Uniform  Resource  Locators 
(URLs)  maintained  within  a  World  Wide  Web  (WWW)  browser,  such  as  Netscape  or 
Mosaic.  A  hotlist  allows  a  subject  to  mark  locations  that  he  or  she  feels  will  be  useful 
for  future  reference.  A  URL  in  a  hotlist  defines  both  the  Internet  address  and  the  type 
of  server  for  a  given  WWW  page.  An  example  of  a  URL  maintained  in  a  hotlist  would 
be  "ftp://www.nps.navy.mil/".  This  is  the  File  Transfer  Protocol  (FTP)  server  residing  on 
"www.nps,navy.mil,"  the  Internet  Protocol(IP)  address  of  the  machine.  In  Table  5,  the 
observer  noted  the  total  number  of  entries  in  each  subject’s  hotlist  and  used  this  to 
provide  the  average  given  for  each  ITM  section. 

Table  5  also  provides  some  statistical  information,  namely  standard  deviation  and 
interval  estimate,  for  this  data.  This  information  was  calculated  using  the  same  method 
discussed  in  detail  in  Chapter  II,  under  "Sample  Size." 
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Yes 

No 

Average 

Hotlist 

sd 

i 

PM33 

5 

- 

26.0 

8.0 

7.84 

PM41 

4 

1 

8.4 

13.1 

12.84 

PM43 

4 

1 

2.8 

2.4 

2.35 

PM51 

1 

4 

0.0 

0.0 

0.0 

Table  5.  Customized  Environment 
sd=standard  deviation 
i=interval  estimate 

Subjects  were  also  asked  if  they  had  access  to  a  Unix  based  computer  account  at 
their  home  or  previous  command  prior  to  arrival  at  the  Naval  Postgraduate  School.  All 
subjects  stated  that  they  did  not  have  previous  access.  Again,  these  were  the  subject’s 
responses  and  were  not  independently  verified  by  the  observer. 

B.  OBSERVATIONS 

The  following  data  were  gathered  during  the  task  observations  conducted  on  ITM 
students  at  the  Naval  Postgraduate  School.  These  groups  of  observations  were  designed 
to  test  the  hypotheses  discussed  in  the  Methodology: 

Hypothesis  1.  The  ability  to  perform  various  tasks  using  the  Internet  and  student 
Unix  accounts  improves  with  experience,  experience  being  defined  as  time  at  the  Naval 
Postgraduate  School. 

Hypothesis  2.  The  prime  source  of  the  students’  ability  to  use  the  Internet  and 
their  Unix  accounts  is  not  through  a  formal  source,  such  as  class  assignments,  computer 
center  lectures,  or  computer  center  personnel,  but  rather  through  an  informal  method  of 
sharing  knowledge  among  themselves. 
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The  first  four  of  the  following  tables  (Tables  6  through  9);  communications,  data 
transfer,  data  location  and  cumulative  results;  are  associated  with  hypothesis  1.  These 
tables  are  based  on  direct  behavioral  observation  by  the  observer,  without  input  from  the 
subject.  The  interval  estimate  for  this  group  of  observations  was  determined  in  Chapter 
II,  under  Sample  Size."  As  discussed  there,  the  sample  data  provided  were  within  53 
seconds  of  the  actual  ITM  population  with  a  confidence  of  0.95.  One  column  on  Tables 
6  through  9  equals  60  seconds.  Each  ITM  section’s  results  for  each  specific  task  are 
contained  in  Appendix  B.  Individual  subject’s  results  are  not  provided.  For  each  of  the 
areas,  the  actual  tasks  used  in  the  observation  are  contained  in  Appendix  A. 


1.  Communications 


Table  6  represents  the  cumulative  totals  for  the  four  observed  tasks  associated  with 
communication.  All  sections  seemed  to  perform  equally  well  on  these  tasks.  As  shown 
in  the  Literature  review,  these  were  the  easiest  tasks  in  the  observation.  Specifically,  these 
communications  tasks  were: 

Al.  Send  an  E-mail  message. 

A2.  Find  a  local  (NPS)  E-mail  address. 

A3.  Find  an  E-mail  address  within  the  same  domain  (.mil). 

A4.  Find  an  E-mail  address  in  an  outside  domain. 


A  domain  in  this  case  is  the  top-level  of  an  e-mail  IP  address,  such  as  .edu  or  .mil. 


PM33 


PM41 


<1 


13 


1-2 


2-3 


3-4 


4-5 


N/A 


10 


PM43 


12 


PM51 


11 


Table  6.  Communication  Tasks  Performance 


2.  Data  Transfer 


Table  7  represents  the  cumulative  totals  for  the  four  observed  tasks  associated  with 
data  transfer.  Performance  stayed  consistent  at  this  point,  except  for  an  improvement  in 
PM33  and  a  decrease  in  PM43.  These  tasks  could  also  be  considered  less  difficult  than 
the  data  location  tasks.  Specifically,  these  data  transfer  tasks  were: 

Bl.  Perform  basic  file  management  (create,  modify  and  delete  files)  within 
subject’s  own  account. 

B2.  Download  a  file  from  a  given  FTP  site. 

B3.  Access  a  given  Gopher  site. 

B4.  Access  a  given  WWW  site. 


<1 

1-2 

2-3 

3-4 

4-5 

N/A 

PM33 

11 

5 

1 

- 

- 

3 

PM41 

6 

5 

3 

- 

- 

6 

PM43 

4 

4 

1 

1 

'  - 

10 

PM51 

5 

5 

- 

- 

- 

10 

Table  7.  Data  Transfer  Tasks  Performance 


3.  Data  Location 

Table  8  represents  the  cumulative  totals  for  the  four  observed  tasks  associated  with 
data  location.  These  were  the  most  difficult  of  the  observed  tasks.  PM33  continued  to 
show  strong  performance,  while  the  other  sections  showed  a  decrease  in  performance. 
This  was  the  task  set  that  gave  PM33  its  overall  edge,  as  shown  in  Table  9.  Specifically, 
these  data  location  tasks  were: 
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Cl.  Find  general  information  on  a  given  topic. 
C2.  Find  a  specific  piece  of  information. 

C3.  Find  a  specific  Gopher  site. 

C4.  Find  a  specific  WWW  site. 


<1 

1-2 

2-3 

3-4 

4-5 

N/A 

PM33 

8 

4 

5 

- 

1 

2 

PM41 

3 

1 

2 

- 

2 

12 

PM43 

2 

3 

2 

- 

- 

13 

PM51 

2 

4 

2 

1 

- 

11 

Table  8.  Data  Location  Tasks  Performance 


4.  Cumulative  Results  for  Hypothesis  1 

Hypothesis  1.  The  ability  to  perform  various  tasks  using  the  Internet  and  student 
Unix  accounts  improves  with  experience,  experience  being  defined  as  time  at  the  Naval 
Postgraduate  School. 

Table  9  shows  the  cumulative  results  of  Tables  6  through  8.  Two  things  appear 
significant  with  this  table.  First,  PM43  performed  poorer  than  all  the  other  sections, 
including  PM51.  Second,  PM33  out  performed  the  other  sections.  These  assumptions 
will  be  examined  in  detail  in  Chapter  IV. 
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Table  9.  Cumulative  Task  Performance 


5.  Knowledge  Source 


Table  10,  knowledge  source,  is  associated  with  the  second  hypothesis: 

Hypothesis  2.  The  prime  source  of  the  students’  ability  to  use  the  Internet  and 
their  Unix  accounts  is  not  through  a  formal  source,  such  as  class  assignments,  computer 
center  lectures,  or  computer  center  personnel,  but  rather  through  an  informal  method  of 
sharing  knowledge  among  themselves. 

Table  10  represents  the  responses  given  by  the  subjects  when  asked  to  rate  the  top 
three  sources  for  the  knowledge  they  displayed  in  performing  the  previous  sets  of  tasks. 
The  responses  were  recorded  as  supplied  by  the  subjects.  They  were  not  independently 
verified  by  the  observer. 

The  weighted  total  in  Table  10  is  given  to  provide  a  more  intuitive  means  of 
comparison  between  the  different  sources  of  knowledge.  The  responses  were  weighted 
as  follows:  Responses  rated  1st  were  given  a  weight  of  3;  responses  rated  2nd  were  given 
a  weight  of  2;  and  responses  rated  3rd  were  given  a  weight  of  1 . 

The  2nd  and  3rd  response  rating  columns  do  not  equal  the  possible  total  of  twenty 
because  several  subjects  responded  with  null  answers. 
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Table  10.  Student’s  Source  of  Knowledge 
(CC  =  Computer  Center) 


Statistical  analysis  and  interpretation  of  all  observation  data  are  provided  in 
Chapter  IV.  Any  raw  data  not  presented  here  are  given  in  Appendix  B. 
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IV.  ANALYSIS  OF  RESULTS 


This  chapter  will  analyze  the  raw  data  reported  in  Chapter  III,  and  determine  if 
it  supports  the  thesis  hypotheses.  It  will  also  look  at  possible  causes  of  any  anomalies 
in  the  data. 

A.  DEMOGRAPHICS 

The  demographics  alone  do  not  provide  support  for  any  of  the  research  questions 
of  this  thesis.  They  do,  however,  draw  attention  to  some  questions  that  arose  during  the 
examination  of  the  data. 

The  most  puzzling  question  that  arose  concerns  the  performance  and  demographics 
of  PM43.  There  seems  to  be  a  correlation  between  Table  4  and  Table  11.  Table  11 
shows  PM43  performing  worse  than  PM51  on  the  cumulative  task  results,  which  seems 
illogical  due  to  PM43’s  supposedly  greater  experience.  Table  4  shows  PM43  with  much 
less  account  usage  than  all  the  sections,  including  PM51.  There  appeared  to  be  some 
systematic  variance  in  the  data. 


<1 

1-2 

2-3 

3-4 

4-5 

N/A 

PM33 

32 

9 

6 

- 

1 

12 

PM41 

18 

6 

6 

- 

2 

28 

PM43 

13 

8 

3 

1 

- 

35 

PM51 

18 

9 

2 

1 

- 

30 

Total 

81 

32 

17 

2 

3 

105 

Table  11.  Cumulative  Task  Performance 
ITM  Section  versus  Time 
(N/A  =  Not  Able  to  Perform  Task) 
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There  were  two  possible  causes  discovered  for  this  problem  with  PM43.  One  is 
a  sampling  error  on  the  part  of  the  observer  and  the  other  is  an  uncontrolled  outside 
influence  upon  PM51,  which  gives  the  mistaken  impression  of  a  problem  in  PM43’s 
performance.  The  sample  taken  was  not  random.  Instead  it  relied  on  volunteers.  In 
the  case  of  PM43,  one  person  facilitated  the  recruitment  of  volunteers  for  the 
observations.  If  this  person  used  acquaintances  with  the  same  social  and  educational 
backgrounds,  and  if  this  background  was  conducive  to  poor  performance  on  the  task 
observations,  then  this  sample  bias  may  be  the  source  of  the  problem. 

The  other  possible  source  of  the  inconsistency  is  a  systematic  variance  in  PM51’s 
data.  In  other  words,  PM43’s  performance  was  not  sub-par,  but  instead  PM51’s 
performance  was  superior.  Upon  investigation,  it  became  known  that  PM5 1  did  receive 
outside  assistance  in  learning  about  the  Internet.  Several  students  from  PM31,  a  previous 
graduated  ITM  section,  remained  on  campus  while  awaiting  orders.  One  of  these 
students,  through  personal  connections,  assisted  PM51  in  getting  started  with  their  Unix 
accounts.  PM51  also  showed  an  increased  response  towards  formal  methods,  as  shown 
in  Appendix  B.  These  factors  combined  may  have  caused  PM51’s  ability  to  increase. 
The  problem,  therefore,  was  not  a  problem  with  PM43,  but  rather  an  over  achievement 
by  PM51.  This  is  believed  to  be  the  actual  cause  of  the  observed  variance  in  the  data. 

The  demographics  showed  no  other  problems  in  the  data.  Account  customization 
was  consistent  with  the  hypothesis  that  the  more  experienced  sections  were  more 
sophisticated  in  there  Internet  usage. 

B.  HYPOTHESIS  1 


Hypothesis  1 .  The  ability  to  perform  various  tasks  using  the  Internet  and  student 
Unix  accounts  improves  with  experience,  experience  being  defined  as  time  at  the  Naval 
Postgraduate  School. 


30 


An  Analysis  of  Variance  (ANOVA)  was  used  to  test  hypothesis  1.  An  ANOVA 
is  used  to  test  if  the  means  of  several  sample  sets  are  statistically  equal.  It  compares  the 
effect  of  different  independent  variables  on  the  dependent  variable.(Emory  &  Cooper, 
1991)  In  this  case,  the  time  taken  by  a  task  was  used  as  the  dependent  variable,  while 
ITM  section  was  the  independent  variable. 

One  problem  involved  with  using  the  ANOVA  was  how  to  represent  the  "Not 
Able"  or  "N/A"  responses  for  the  subjects  who  took  longer  than  five  minutes.  "N/A" 
must  somehow  be  presented  as  interval  data.  There  seemed  to  be  two  possible  ways  to 
do  this. 

One  option  would  be  to  give  "N/A"  an  arbitrary  numerical  value  based  on  some 
quantity  such  as  mean  time  for  receiving  help  from  a  computer  center  consultant.  The 
problem  with  this  method  was  determining  an  equitable  numerical  value  to  assign  to 
"N/A."  It  was  felt  that  the  "N/A"  response  could  be  any  length  of  time  from  slightly 
more  than  five  minutes  to  possibly  indefinite.  This  option  was  deemed  unreliable  and 
unscientific. 

The  second  option  was  based  on  the  bipolar  distribution  of  the  data.  As  shown 
in  Table  12,  most  measurable  responses  occurred  early  in  the  task  observation.  This 
shows  that  for  most  subjects,  they  either  could  perform  the  task  or  they  could  not.  In 
other  words,  they  were  not  able  to  work  through  the  task  if  they  did  not  immediately 
know  the  procedure.  This  dichotomy  presented  the  opportunity  to  assign  integer  values 
to  whether  the  subject  was  able  to  perform  the  task  or  not. 


<1 

1-2 

2-3 

3-4 

4-5 

N/A 

Total 

81 

32 

17 

2 

3 

105 

Table  12.  Cumulative  Response  Time 
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Therefore,  for  the  ANOVA  test  of  hypothesis  1  presented  in  Table  13,  subjects 
who  were  able  to  perform  the  tasks  were  assigned  a  value  of  one.  Subjects  who  were 
unable  to  perform  the  task  were  assigned  a  value  of  zero. 

The  null  hypothesis(H0)  used  to  check  hypothesis  1  is  that  the  task  performance 
of  each  section  is  statistically  equal  to  the  task  performance  of  the  other  sections.  Table 
13  shows  the  results  of  the  ANOVA  for  each  task  category  plus  for  the  cumulative. 

Null  Hypothesis: 

Ho-  f>33=f>41=F>43'=^>51 

where:  P  =  performance  on  given  task  set 

subscript  =  ITM  section 


Tasks 

Sum  of 

DF 

Mean 

F 

Sig  of 

Squares 

Squares 

F 

Communications 

.650 

3 

.217 

.853 

.469 

Data  Transfer 

1.737 

3 

.579 

2.628 

.056 

Data  Location 

3.850 

3 

1.283 

6.058 

.001 

Cumulative 

4.946 

3 

1.649 

7.190 

.000 

Table  13.  ANOVA  of  Task  Results 


Using  p<.05  as  the  cutoff  point,  we  failed  to  reject  the  null  hypothesis  in  the  case 
of  communications  and  data  transfer.  This  means  that  all  four  ITM  sections  performed 
statistically  equal  on  these  tasks.  In  the  case  of  data  location,  however,  we  did  reject  the 
null  hypothesis,  showing  that  the  different  ITM  sections  were  not  equal. 

The  tools  used  with  information  location  are  more  complex  than  those  used  for 
communications  and  data  transfer.  This  is  consistent  with  failing  to  reject  the  null 
hypothesis  for  communications  and  data  transfer,  but  rejecting  it  for  data  location. 
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We  also  rejected  the  null  hypothesis  for  the  cumulative  data.  While  this  supports 
our  hypothesis,  the  cumulative  calculations  also  had  the  advantage  of  a  data  set  three 
times  as  large  as  the  non-cumulative  tasks.  The  increased  size  pf  the  data  set  made  even 
minor  variations  statistically  significant. 

C.  HYPOTHESIS  2 

Hypothesis  2.  The  prime  source  of  the  students’  ability  to  use  the  Internet  and 
their  Unix  accounts  is  not  through  a  formal  source,  such  as  class  assignments,  computer 
center  lectures,  or  computer  center  personnel,  but  rather  through  an  informal  method  of 
sharing  knowledge  among  themselves. 

Statistical  analysis  was  performed  on  the  data  using  a  Friedman  Two-Way 
ANOVA.  A  Friedman  Two-Way  ANOVA  examines  the  variance  of  ranked  data  within 
each  block.  A  block  in  this  case  was  one  subject.  It  is  based  on  two  assumptions.  One, 
that  the  results  of  one  subject  do  not  influence  any  other  subject.  Two,  that  each 
subject’s  data  is  arranged  in  a  rank  order.  Both  of  these  criteria  were  met  by  the 
observation  data  for  hypothesis  2.  (Conover,  1971) 

Again,  there  was  a  problem  with  null  data.  The  subjects  were  asked  to  rank  only 
their  top  three  choices  out  of  a  possible  six  choices.  It  was  decided  to  give  all  null  data 
the  rank  of  four.  Therefore,  each  block  of  data  would  have  the  top  three  sources,  as 
ranked  by  the  subjects,  and  all  remaining  options  would  be  given  the  rank  of  four.  The 
Friedman  Two-Way  ANOVA  is  tolerant  of  ties  in  the  ranked  data.  (Conover,  1971) 

The  Friedman  Two-Way  ANOVA  provides  two  pieces  of  information.  One,  a 
mean  rank  for  each  variable,  and  two,  a  significance  factor  for  testing  a  null  hypothesis 
of  equality. 

The  null  hypothesis  used  for  hypothesis  2  was  that  all  sources  of  ability  were 
ranked  equal.  Table  14  shows  the  results  of  the  Friedman  Two-Way  ANOVA  testing  the 
following  null  hypothesis. 
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Null  Hypothesis  (Ho): 


Hq.  82—82—83—84—85—85 

where:  subscript  =  specific  source  as  shown  in  Table  15. 


x2 

DF 

Sig 

32.6857 

5 

.000 

Table  14.  Friedman  Two-Way  ANOVA  Results 


Using  a  p<.05  cutoff,  we  were  able  to  reject  the  null  hypothesis.  Therefore,  the 
sources  that  subjects  reported  for  where  they  learned  the  Internet  were  not  statistically 
equal. 

The  rest  of  the  information  provided  by  the  Friedman  Two-Way  ANOVA  is  given 
in  Table  15.  This  gives  us  the  mean  rank  of  the  data  over  all  subjects. 


Sn 

Source 

Mean 

Rank 

1 

CC 

Personnel 

4.47 

2 

CC  Class 

4.47 

3 

CC 

Handout 

4.32 

4 

On  Own 

2.15 

5 

Classmates 

2.40 

6 

Assignment 

3.17 

- 

Other 

- 

Table  15.  Mean  Rank  of  Knowledge  Sources 
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Overall,  this  shows  that  "On  Own"  and  "Shown  by  Classmates"  are  the  common 
source  of  learning  Internet  techniques.  These  can  be  considered  the  informal  techniques, 
thus  supporting  the  hypothesis  that  students  are  learning  in  informal  ways.  The  only 
formal  method  worth  mentioning  is  "class  assignments"  which  performed  far  better  than 
the  Computer  Center  related  sources.  The  results  of  each  individual  section  are  shown  in 
Appendix  C. 

The  one  exception  is  PM51’s  individual  results,  as  shown  in  Appendix  C,  which 
show  a  stronger  use  of  formal  methods.  This  is  the  only  section  which  did  not  rank  "on 
own"  and  "shown  by  classmates"  as  the  top  two  sources.  PM51,  instead,  ranked  "class 
assignments"  as  their  top  choice.  This  is  assumed  to  be  the  result  of  more  classroom 
instructors  beginning  to  including  the  Internet  in  their  class  materials. 

The  consequences  of  these  observations  will  be  discussed  in  Chapter  V  , 
"Conclusions  and  Recommendations." 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

1.  Study  Benefits 

This  thesis  has  portrayed  the  current  state  of  Internet  usage  by  ITM  students  at  the 
Naval  Postgraduate  School.  ITM  students  are  expected  to  function  as  the  future  DoD  IT 
managers.  In  this  role  they  will  need  to  be  literate  in  the  use  of  the  Internet.  It  has 
shown  the  need  for  earlier  direction  in  teaching  the  Internet  to  ITM  students. 

As  a  secondary  benefit,  the  data  recorded  in  this  thesis  may  also  be  useful  in 
future  research  as  both  a  starting  point  and  a  baseline  for  comparison  against  new  data. 

2.  Research  Questions 

Is  there  a  relationship  between  experience  and  the  ability  to  use  the  Internet 
effectively?  The  answer  is  a  definite  yes.  This  in  itself  is  not  remarkable,  and  in  fact 
seems  quite  obvious.  The  interesting  revelation  is  the  characteristics  of  the  relationship. 

Only  PM33’s  performance  could  be  considered  remotely  adequate,  and  even  PM33 
was  only  able  to  perform  48  out  60  tasks,  or  80%.  The  other  sections  performance 
should  be  considered  substandard.  This  is  not  necessarily  a  major  problem.  Students,  as 
shown  by  PM33,  are  becoming  Internet  literate  before  they  leave  NPS.  They  are  not, 
however,  receiving  the  full  benefit  of  Internet  usage  during  their  entire  course  of  study 
at  NPS. 

What  instructional  source  has  taught  ITM  students  to  use  the  Internet?  The  overall 
source  of  instruction  is  very  informal.  There  is  certainly  a  correlation  between  this  fact 


37 


and  the  low  scores  show  by  all  sections  except  PM33.  Informal  learning  methods  are 
likely  to  be  more  inefficient  than  formal  ones. 

There  is  however,  as  shown  in  Table  16,  a  migration  towards  more  formal 
methods  taking  place.  This  should  be  considered  a  positive  sign  as  shown  by  PM51’s 
advanced  performance. 


S„ 

Source 

PM33 

PM41 

PM43 

PM51 

1 

CC  Personnel 

4.60 

4.00 

4.70 

4.60 

2 

CC  Class 

4.60 

4.80 

3.90 

4.60 

3 

CC  Handout 

4.20 

4.20 

4.70 

4.20 

4 

On  Own 

1.20 

2.00 

2.50 

3.20 

5 

Classmates 

2.20 

2.40 

2.00 

2.60 

6 

Assignment 

4.20 

3.60 

3.20 

1.80 

Table  16.  Source  Rank  by  Section 


B.  RECOMMENDATIONS 

The  following  recommendations  are  designed  to  improve  the  ability  of  ITM 
students  to  use  the  Internet,  based  on  the  results  of  the  research  questions. 

Based  on  the  research  conducted  in  this  thesis,  formal  classroom  instruction  is 
needed  to  teach  ITM  students  the  Internet.  This  conclusion  is  based  on  the  current 
informal  methods  used  and  the  low  observation  performance  they  produced.  If  a  formal 
class  is  impracticable,  then  an  online  tutorial  may  be  appropriate.  The  advantages  and 
disadvantages  of  both  options  should  be  examined  in  detail  as  possible  future  research. 

Currently,  the  majority  of  the  students  learn  through  informal  methods.  Non¬ 
mandatory  computer  center  classes  are  available  to  all  these  students.  They  are  not  being 
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used.  If  this  status  quo  continues,  without  the  adoption  of  a  formal  Internet  class,  then 
an  online  tutorial  would  be  the  appropriate  instructional  tool.  Stimulating  usage  will  most 
likely  continue  to  be  a  drawback  with  any  non-mandatory  method. 

However,  it  has  also  been  shown  that  there  is  a  gradual  shift  taking  place  towards 
formal  instruction  as  shown  by  PM51’s  results.  If  it  is  desired  for  this  trend  to  continue, 
then  a  formal  Internet  class  would  be  the  appropriate  solution.  A  formal  class  would 
follow  a  standard  teaching  methodology,  which  would  provide  a  much  greater  chance  of 
long  term  comprehension  than  informal  methods. 

1.  Online  Tutorial 

Constructing  an  online  tutorial  should  almost  certainly  be  considered.  Even  if  a 
formal  class  structure  is  adopted,  a  ready  reference  source  would  be  invaluable  until  the 
students  became  more  sophisticated  in  customizing  there  own  environment  and  tools. 

The  most  useful  form  for  a  tutorial  would  be  a  WWW  based  page  on  a  server 
controlled  by  the  ITM  curriculum,  or  at  least  by  the  Systems  Management  curriculum. 

An  online  tutorial  also  has  the  benefit  of  portability.  With  the  increased  use  of 
the  Internet  within  DoD,  this  could  have  several  advantages.  A  well  designed  tutorial 
could  be  exported  to  other  curriculums  and  DoD  activities  with  WWW  capabilities.  Also, 
if  there  were  any  interest  in  standardizing  what  resource  discovery  tools  and  services  were 
used  within  DoD,  a  canned  tutorial  could  be  a  way  to  publicize  them. 

2.  Class  Instruction 

The  preferred  means  of  teaching  the  Internet,  however,  would  be  the  formal 
classroom  instruction.  A  basic  Internet  skills  curriculum  would  not  occupy  enough 
material  to  fill  an  entire  quarter  course.  It  would  be  better  taught  as  a  portion  of  an 


39 


introductory  course  such  as  IS2000  or  as  an  independent  half  quarter  course  during  the 
refresher  quarter. 

Formal  classroom  instruction  also  has  the  advantage  of  being  mandatory. 
Hopefully,  mandatory  instruction  will  not  be  necessary  once  the  student  sees  the 
advantages  of  using  the  Internet,  but  until  that  time,  class  assignments  would  point  the 
student  in  the  right  direction. 

C.  FURTHER  RESEARCH 


Based  on  the  above  recommendations,  follow-on  research  into  designing  a  tutorial 
or  class  may  be  useful.  Further  testing  to  verify  the  results  of  these  implementations 
would  then  also  be  useful. 

1 .  Design  the  online  tutorial  to  be  used.  This  would  require  a  strong 
familiarization  with  hypertext  mark-up  language  (html)  and  its  associated  use  on  WWW 
servers.  It  would  also  require  research  in  tutorial  design  methods.  Finally,  the  material 
for  constructing  the  tutorial  would  have  to  be  gathered  and  verified. 

2.  Design  the  class  syllabus  for  an  Internet  course.  Again,  this  would  require  the 
gathering  and  verification  of  the  class  material.  This  material  would  then  have  to  be 
placed  into  a  logical  sequence,  and  the  associated  teaching  media  developed. 

3.  Test  the  effectiveness  of  whatever  new  techniques  are  implemented.  This  would 
consist  of  reconducting  the  observations  of  this  thesis.  These  new  results  would  be 
compared  to  this  thesis’s  baseline  results.  This  would  confirm  the  effectiveness  of 
whatever  new  instruction  methods  were  chosen.  Even  if  no  new  methods  were 
implemented,  tracking  the  growing  use  of  the  Internet  in  other  ITM  classes  would  at  least 
confirm  the  results  shown  by  PM51. 
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APPENDIX  A.  OBSERVATION  TASKS 


Al.  Send  E-mail  to  the  following  address:  afmusgro@nps.navy.mil 
Subject:  Test  Body:  This  is  a  test. 

A2.  William  J.  Haga  is  a  faculty  member  at  NPS,  find  his  e-mail  address. 

A3.  Bill  Doe  is  in  the  Army,  location  unknown,  find  his  e-mail  address. 

A4.  Karen  Jones  works  at  the  University  of  Alaska  at  Fairbanks,  find  her  e-mail  address. 


Bl.  Create  a  file  "test",  move  it  to  any  other  directory,  then  delete  it. 

B2.  Download  the  file  "newnames.txt:  from  the  following  FTP  site: 

site:  byrd.mu.wvnet.edu  login:  anonymous 

directory:  pub/history /military /USN 

B3.  Access  the  following  gopher  site: 
gopher.ucr.ac.cr 

B4.  Access  the  following  WWW  page: 
http://wsk.eit.com/ 

Cl.  Find  information  on  the  subject  "alienation". 

C2.  Find  a  copy  of  Sir  Walter  Scott’s  book  Ivanhoe. 

C3.  Find  the  following  gopher  site: 

PATHY  Gopher  for  Medical  Information 

C4.  Find  the  following  WWW  page: 

EUNet  Norway  Information  Services 

For  the  above  tasks,  where  did  you  gain  your  knowledge  (rank  the  top  three): 

a.  Shown  by  Computer  Center  Personnel 

b.  Attended  Computer  Center  Class 

c.  Read  Computer  Center  Handout/Bulletin 

d.  On  your  own/exploring 

e.  Shown  by  classmates 

f.  From  class  assignments  in  a  required  course 

g.  Other  (please  specify) 
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APPENDIX  B.  OBSERVATION  RAW  DATA 


Demographics 

Hotlist  Size  by  Subject  and  Section 


AVG 
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3 

2.8 
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0 
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0 
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Task  Observations 

For  ease  of  visual  examination  all  "0"  responses  have  been  replaced  with  a 
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- 

- 
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Communications  (continued) 

A2.  Find  a  local  E-mail  (NPS)  address. 
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A3.  Find  an  E-mail  address  within  the  same  domain  (  mil) 
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A4.  Find  an  E-mail  address  in  an  outside  domain. 
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Data  Retrieval 


B 1 .  Perform  basic  file  management. 


B2.  Download  a  file  from  a  given  FTP  site. 

<1  1-2  2-3  3-4  4-5  N/A 


PM33  -  4  1 


PM41  -  2  -  -  -  3 


PM43  1  1  1  -  -  2 


PM51  4  -  1 


B3.  Access  a  given  Gopher  site. 
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Data  Retrieval  (continued) 

B4.  Access  a  given  WWW  site. 
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Data  Location 

Cl.  Find  general  information  on  a  given  topic. 
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C2.  Find  a  specific  piece  of  information. 
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C3.  Find  a  specific  Gopher  site. 
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C4.  Find  a  specific  WWW  site. 
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Source  of  Knowledge  by  Section 


CC  =  Computer  Center 
Weighted  Total:  1st  =  3 

2nd  =  2 
3rd  =  1 
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Source  of  Knowledge  by  Section  (continued) 
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Handout 


On  Own 


Classmates 


Assignment 


Other 


Weighted 

Total 
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APPENDIX  C.  FRIEDMAN  TWO-WAY  ANOVA  RESULTS 

Mean  Ranks  by  Section 
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PM43  (Continued) 


Sn 

Source 

Mean 

Rank 

1 

CC  Personnel 

4.70 

2 

CC  Class 

3.90 

3 

CC  Handout 

4.70 

4 

On  Own 

2.50 

5 

Classmates 

2.00 

6 

Assignment 

3.20 

- 

Other 

- 

PM51 


t 

DF 

Sig 

9.5714 

5 

.0883 

Sn 


1 

2 

3_ 

4 

5 


Source 


CC  Personnel 
CC  Class 
CC  Handout 
On  Own 
Classmates 
Assignment 


Mean 

Rank 

4.60 

4.60 

4.20 

3.20 

2.60 


6 


1.80 


54 


LIST  OF  REFERENCES 


Alessi,  Stephan  M.  &  Trollop,  Stanley  R.,  Computer-Based  Instruction,  Methods  and 
Development ,  Prentice  Hall,  1991. 

Conover,  W.  J.,  Practical  Nonparametric  Statistics,  John  Wiley  &  Sons,  Inc.,  1971. 

Dillon,  Ronna  F.  &  Sternberg,  Robert  J.,  Cognition  and  Instruction,  Academic  Press, 
1986. 

Emory,  William  C.  &  Cooper,  Donald  R.,  Business  Research  Methods,  Irwin,  Inc., 
1991. 

Foster,  Steven  &  Barrie,  Fred,  Veronica  FAQ,  gopher://veronica.  scs.unr.edu/veronica, 
1993. 

Ngwenyama,  Ojelanki  K.,  "Developing  End-Users’  Systems  Development  Competence: 
An  Exploratory  Study,"  Information  &  Management,  vol.  25,  pp.  291-302,  December 
1993. 

Obraczka,  Katia,  Danzig,  Peter  B.,  &  Li,  Shih-Hao,  "Internet  Resource  Discovery 
Services,"  Computer,  vol.  26,  pp.  8-22,  September  1993. 

Perry,  Tekla  S.,  "E-Mail  at  Work,"  IEEE  Spectrum,  vol.  29,  pp.  24-28,  October  1992. 

Pinkerton,  Brian,  Technical  Description  of  the  WebCrawler, 
http://webcrawler.cs.washington.edU/W ebCrawler/W  ebCrawler.html,  1 995 . 

Postel,  Jonathan  B.,  Simple  Mail  Transfer  Protocol,  Information  Sciences  Institute, 
University  of  Southern  California,  1982. 

Rovers,  Perry,  Anonymous  FTP  FAQ,  http://www.mid.net/FTP/,  1994. 

Schwartz,  Michael  F.,  "Internet  Resource  Discovery  at  the  University  of  Colorado," 
Computer,  vol.  26,  pp.  25-35,  September  1993. 

Watson,  Richard,  "Creating  and  Sustaining  a  Global  Community  of  Scholars,"  MIS 
Quarterly,  September  1994. 

Whitmore,  Neter  Wasserman,  Applied  Statistics,  Allyn  and  Bacon,  1993. 


55 


56 


BIBLIOGRAPHY 


Huynh,  Minh,  Popkin,  Laird,  &  Stecker,  Matthew,  Constructing  a  Corporate  Memory 
Infrastructure  from  Internet  Discovery  Technologies,  Marble  Associates,  Inc.,  1994. 

Martin,  J.L.,  "Travels  with  Gopher,"  Computer,  vol.  26,  pp.  84-87,  May  1993. 

Rosenfield,  Louis,  Clearinghouse  for  Subject-Oriented  Internet  Resource  Guides, 
http://asa.ugi.lib.umich.edu/chhome.html/,  1994. 

Schatz,  Bruce  R.  &  Hardin,  Joseph  B.,  "NCSA  Mosaic  and  the  World  Wide  Web:  Global 
Hypermedia  Protocols  for  the  Internet,"  Science,  vol.  265,  12  August  1994. 

Travers,  Robert  M.W.,  Essentials  of  Learning:  An  Overview  for  Students  of  Education, 
Macmillan  Co.,  1963. 

Van  Vliet,  Paul  J.A.,  Kletke,  Marilyn  G.,  &  Chakraborty,  Goutam,  "The  Measurement  of 
Computer  Literacy:  a  Comparison  of  Self-Appraisal  and  Objective  Tests,"  International 
Journal  of  Human-Computer  Studies,  vol.  40,  pp.  835-57,  May  1994. 

Vetter,  Ronald  J.,  Spell,  Chris,  &  Ward,  Charles,  "Mosaic  and  the  World-Wide  Web," 
Computer,  vol.  27,  pp.  49-57,  October  1994. 


57 


58 


INITIAL  DISTRIBUTION  LIST 


1.  Defense  Technical  Information  Center  ............  2 

Cameron  Station 

Alexandria,  Virginia  22304-6145 

2.  Library,  Code  52 . 2 

Naval  Postgraduate  School 

Monterey,  California  93943-5101 

3.  William  J.  Haga,  Code  Sm/Hg . 1 

Naval  Postgraduate  School 

Monterey,  California  93943-5001 

4.  William  B.  Short,  Code  Sm/Sh  . 1 

Naval  Postgraduate  School 

Monterey,  California  93943-5001 

5.  Lt.  Albert  F.  Musgrove,  USN  .  . . 2 

416  Warley  Street 

Portsmouth,  Rhode  Island  02871 


59 


